Analyses
Pilot analyses
282 people responded. We may want to further screen these 282 responses for short response durations. According to communications with Qualtrics in early December, this is the number of seconds to complete the entire survey. Currently, the “number of people who clicked on the assessment link (330)” versus valid n (282) may take care of our very low duration respondents. The shortest response duration in the 330 datafile is 101 whereas the lowest in the 282 datafile is 43. All analyses were performed via R version R version 4.1.0 (2021-05-18) [@R-base].
Classical test theory
In addition to the below interactive plot [via plotly version 4.9.3; @R-plotly], a full inter-item correlation matrix is located in Appendix @ref(corrs).
## Some items ( I find it difficult to mentally disconnect from work. ) were negatively correlated with the total scale and
## probably should be reversed.
## To do this, run the function again with the 'check.keys=TRUE' optionInternal consistency estimates
Condition 1 administered items within the substantive dimensions (with successive randomized blocks of Cognitive, Affective, and Behavioral items). Condition 2 administered items within the attitudinal dimensions (with successive randomized blocks of Absorption, Vigor, and Dedication items). Condition 3 stressed the substantive dimensions (with items fully randomized regardless of attitudinal association). Condition 4 stressed the attitudinal dimensions (with items fully randomized within attitudinal dimension regardless of substantive scale association, see Chapter @ref(conds) and Appendix @ref(pilot2)). All internal consistency estimates were generated via psych version 2.1.3 [@R-psych]. Alphas for the candidate 12-item scales were:
| Dimension | Undifferentiated | Condition 1 | Condition 2 | Condition 3 | Condition 4 |
|---|---|---|---|---|---|
| Affective | 0.87 | 0.88 | 0.84 | 0.88 | 0.87 |
| Behavioral | 0.79 | 0.73 | 0.84 | 0.76 | 0.87 |
| Cognitive | 0.81 | 0.79 | 0.85 | 0.8 | 0.87 |
| Absorption | 0.74 | 0.72 | 0.77 | 0.72 | 0.87 |
| Vigor | 0.85 | 0.8 | 0.88 | 0.84 | 0.87 |
| Dedication | 0.9 | 0.9 | 0.91 | 0.91 | 0.87 |
“Cell” level alphas (4 items each scale, responses collapsed across administrative conditions) were:
| Cell | Alpha |
|---|---|
| Affective - Absorption | 0.66 |
| Affective - Vigor | 0.71 |
| Affective - Dedication | 0.75 |
| Behavioral - Absorption | 0.56 |
| Behavioral - Vigor | 0.7 |
| Behavioral - Dedication | 0.64 |
| Cognitive - Absorption | 0.59 |
| Cognitive - Vigor | 0.62 |
| Cognitive - Dedication | 0.83 |
Corrected item-total correlations are presented in Appendix @ref(rdrops)
Confirmatory factor analyses
We used lavaan version 0.6.8 [@R-lavaan] and semPlot version 1.1.2 [@R-semPlot]
| Model | \(\chi^2\) | df | RMSEA | SRMR | CFI | TLI | AIC |
|---|---|---|---|---|---|---|---|
| 3-factor substantive | 2159.21 | 591 | 0.11 | 0.1 | 0.64 | 0.62 | 25481.97 |
| Hierarchical substantive | 2159.21 | 591 | 0.11 | 0.1 | 0.64 | 0.62 | 25481.97 |
| 3-factor attitudinal | 2318.92 | 591 | 0.11 | 0.1 | 0.6 | 0.58 | 25641.68 |
| Hierarchical attitudinal | 2318.92 | 591 | 0.11 | 0.1 | 0.6 | 0.58 | 25641.68 |
Multigroup analyses
- using experimental condition as “multiple groups” (measurement invariance)
Bifactor analyses
bifactor analysis are most commonly applied in the exploration of common method variance [see, for example, @reise_rediscovery_2012; @rodriguez_evaluating_2016]. @giordano_exploratory_2020 provide an overview regarding past and potential applications of exploratory bifactor analysis and cite @reise_rediscovery_2012 as an influential impetus for the resurgence of bifactor models in general.
## lavaan 0.6-8 did NOT end normally after 820 iterations
## ** WARNING ** Estimates below are most likely unreliable
##
## Estimator ML
## Optimization method NLMINB
## Number of model parameters 114
##
## Used Total
## Number of observations 239 282
##
## Model Test User Model:
##
## Test statistic NA
## Degrees of freedom NA
##
## Parameter Estimates:
##
## Standard errors Standard
## Information Expected
## Information saturated (h1) model Structured
##
## Latent Variables:
## Estimate Std.Err z-value P(>|z|)
## Cognitive =~
## Item_1 1.000
## Item_2 0.516 NA
## Item_3 1.090 NA
## Item_4 0.353 NA
## Item_13 1.130 NA
## Item_14 0.838 NA
## Item_15 0.798 NA
## Item_16 1.098 NA
## Item_25 0.988 NA
## Item_26 0.909 NA
## Item_27 1.080 NA
## Item_28 0.999 NA
## Affective =~
## Item_5 1.000
## Item_6 0.656 NA
## Item_7 0.475 NA
## Item_8 1.242 NA
## Item_17 0.933 NA
## Item_18 1.372 NA
## Item_19 1.044 NA
## Item_20 0.536 NA
## Item_29 0.841 NA
## Item_30 0.381 NA
## Item_31 0.724 NA
## Item_32 1.063 NA
## Behavioral =~
## Item_9 1.000
## Item_10 1.109 NA
## Item_11 0.572 NA
## Item_12 1.176 NA
## Item_21 1.362 NA
## Item_22 1.573 NA
## Item_23 0.873 NA
## Item_24 0.890 NA
## Item_33 0.707 NA
## Item_34 0.879 NA
## Item_35 1.237 NA
## Item_36 0.369 NA
## Absorption =~
## Item_1 1.000
## Item_2 19131.383 NA
## Item_3 -390.734 NA
## Item_4 21074.473 NA
## Item_5 2937.496 NA
## Item_6 -1569.896 NA
## Item_7 727.592 NA
## Item_8 -1533.787 NA
## Item_9 4326.028 NA
## Item_10 5122.173 NA
## Item_11 -1534.018 NA
## Item_12 -1356.265 NA
## Vigor =~
## Item_13 1.000
## Item_14 -3.413 NA
## Item_15 0.449 NA
## Item_16 -0.517 NA
## Item_17 -0.246 NA
## Item_18 -1.152 NA
## Item_19 0.766 NA
## Item_20 -3.510 NA
## Item_21 0.381 NA
## Item_22 -0.278 NA
## Item_23 0.324 NA
## Item_24 0.093 NA
## Dedication =~
## Item_25 1.000
## Item_26 1.378 NA
## Item_27 1.009 NA
## Item_28 0.830 NA
## Item_29 0.885 NA
## Item_30 0.650 NA
## Item_31 0.274 NA
## Item_32 0.512 NA
## Item_33 0.010 NA
## Item_34 0.122 NA
## Item_35 0.888 NA
## Item_36 0.783 NA
##
## Covariances:
## Estimate Std.Err z-value P(>|z|)
## Cognitive ~~
## Affective 0.549 NA
## Behavioral 0.363 NA
## Affective ~~
## Behavioral 0.372 NA
## Absorption ~~
## Vigor 0.000 NA
## Dedication -0.000 NA
## Vigor ~~
## Dedication -0.082 NA
## Cognitive ~~
## Absorption 0.000
## Vigor 0.000
## Dedication 0.000
## Affective ~~
## Absorption 0.000
## Vigor 0.000
## Dedication 0.000
## Behavioral ~~
## Absorption 0.000
## Vigor 0.000
## Dedication 0.000
##
## Variances:
## Estimate Std.Err z-value P(>|z|)
## .Item_1 1.343 NA
## .Item_2 0.350 NA
## .Item_3 0.786 NA
## .Item_4 0.124 NA
## .Item_13 0.809 NA
## .Item_14 0.659 NA
## .Item_15 1.471 NA
## .Item_16 0.760 NA
## .Item_25 1.350 NA
## .Item_26 0.629 NA
## .Item_27 1.282 NA
## .Item_28 0.763 NA
## .Item_5 1.324 NA
## .Item_6 1.195 NA
## .Item_7 0.845 NA
## .Item_8 0.688 NA
## .Item_17 0.484 NA
## .Item_18 0.559 NA
## .Item_19 0.771 NA
## .Item_20 0.780 NA
## .Item_29 0.479 NA
## .Item_30 1.514 NA
## .Item_31 0.538 NA
## .Item_32 0.667 NA
## .Item_9 1.293 NA
## .Item_10 1.961 NA
## .Item_11 1.288 NA
## .Item_12 1.342 NA
## .Item_21 0.894 NA
## .Item_22 0.510 NA
## .Item_23 0.338 NA
## .Item_24 1.026 NA
## .Item_33 0.401 NA
## .Item_34 0.532 NA
## .Item_35 0.547 NA
## .Item_36 0.984 NA
## Cognitive 0.593 NA
## Affective 0.639 NA
## Behavioral 0.283 NA
## Absorption 0.000 NA
## Vigor 0.072 NA
## Dedication 0.596 NACurrently pretty messed up prior to item deletion (Vigor especially). Model also technically not converging need Renata to fix.
Deese guys also do bifactor stuff: @mansolf_when_2017
Summary
Recommendation for final instrument based on consideration of all of the above pieces of evidence